Question: Are soccer referees more likely to give red cards to dark-skin-toned players than light-skin-toned players?¶
-> GridSpec: Specifies the geometry of the grid that a subplot can be placed in.¶
-> tqdm: To show the progress¶
-> missingno: provides a small toolset of flexible and easy-to-use missing data visualizations and utilities that allows you to get a quick visual summary of the completeness (or lack thereof) of your dataset¶
In [1]:
%matplotlib inline
import matplotlib as mpl
from matplotlib import pyplot as plt
from matplotlib.pyplot import GridSpec
import seaborn as sns
import numpy as np
import pandas as pd
import os, sys
from tqdm import tqdm
import warnings
warnings.filterwarnings('ignore')
# sns.set_context("poster", font_scale=1.3)
import missingno as msno
import pandas_profiling
from sklearn.datasets import make_blobs
import time
from pandas.plotting import scatter_matrix
DATA:¶
| Variable Name: | Variable Description: |
|---|---|
| playerShort | short player ID |
| player | player name |
| club | player club |
| leagueCountry | country of player club (England, Germany, France, and Spain) |
| height | player height (in cm) |
| weight | player weight (in kg) |
| position | player position |
| games | number of games in the player-referee dyad |
| goals | number of goals in the player-referee dyad |
| yellowCards | number of yellow cards player received from the referee |
| yellowReds | number of yellow-red cards player received from the referee |
| redCards | number of red cards player received from the referee |
| photoID | ID of player photo (if available) |
| rater1 | skin rating of photo by rater 1 |
| rater2 | skin rating of photo by rater 2 |
| refNum | unique referee ID number (referee name removed for anonymizing purposes) |
| refCountry | unique referee country ID number |
| meanIAT | mean implicit bias score (using the race IAT) for referee country |
| nIAT | sample size for race IAT in that particular country |
| seIAT | standard error for mean estimate of race IAT |
| meanExp | mean explicit bias score (using a racial thermometer task) for referee country |
| nExp | sample size for explicit bias in that particular country |
| seExp | standard error for mean estimate of explicit bias measure |
In [2]:
#Read the data
data_df = pd.read_csv('../data/redcard.csv')
data_df.head()
Out[2]:
In [3]:
data_df.shape
Out[3]:
In [4]:
data_df.describe().T
Out[4]:
In [5]:
data_df.dtypes
Out[5]:
In [6]:
columns = data_df.columns.tolist()
columns
Out[6]:
Questions on the dataset¶
How do we operationalize the question of referees giving more red cards to dark skinned players?
- Counterfactual: if the player were lighter, a ref is more likely to have given a yellow or no card for the same offense under the same conditions
- Regression: accounting for confounding, darker players have positive coefficient on regression against proportion red/total card
Potential issues
- How to combine rater1 and rater2? Average them? What if they disagree? Throw it out?
- Is data imbalanced, i.e. red cards are very rare?
- Is data biased, i.e. players have different amounts of play time? Is this a summary of their whole career?
- How do I know I've accounted for all forms of confounding?
First, is there systematic discrimination across all refs?
Exploration/hypotheses:
- Distribution of games played
- red cards vs games played
- Reds per game played vs total cards per game played by skin color
- Distribution of # red, # yellow, total cards, and fraction red per game played for all players by avg skin color
- How many refs did players encounter?
- Do some clubs play more aggresively and get carded more? Or are more reserved and get less?
- Does carding vary by leagueCountry?
- Do high scorers get more slack (fewer cards) for the same position?
- Are there some referees that give more red/yellow cards than others?
- how consistent are raters? Check with Cohen's kappa.
- how do red cards vary by position? e.g. defenders get more?
- Do players with more games get more cards, and is there difference across skin color?
- indication of bias depending on refCountry?
Data Cleaning¶
Create multiple tables for
- players
- refrees
- clubs
- countries
- dyands (player-refree combo)
Players¶
In [7]:
players_index = 'playerShort'
players_cols = [
'player',
'birthday',
'height',
'weight',
'position',
'photoID',
'rater1',
'rater2',
]
In [8]:
#Check if we have duplicate entries
#If the same player has different values in differnt rows, then the cell value will be > 1
all_cols_unique_players = data_df.groupby('playerShort').agg({col:'nunique' for col in players_cols})
all_cols_unique_players.head()
Out[8]:
In [9]:
all_cols_unique_players[all_cols_unique_players > 1].dropna().shape[0] == 0
Out[9]:
This being True says that all the values are same per player across multiple rows¶
In [10]:
def get_subgroup(dataframe, g_index, g_columns):
'''
purpose: Group the dataframe based on index and check for the group's uniqueness
parameters:
dataframe: DF on which grouping and uniqueness test needs to be done
g_index: Index for grouping
g_columns: Columns for grouping
return:
Dataframe with unique entries and max value for each column
'''
g = dataframe.groupby(g_index).agg({col:'nunique' for col in g_columns})
if g[g > 1].dropna().shape[0] != 0:
print("This group doesn't have unique values")
return dataframe.groupby(g_index).agg({col:'max' for col in g_columns})
In [11]:
players = get_subgroup(data_df,players_index, players_cols)
players.head()
Out[11]:
Club¶
In [12]:
club_index = 'club'
club_cols = ['leagueCountry']
clubs = get_subgroup(data_df,club_index,club_cols)
clubs.head()
Out[12]:
In [13]:
clubs.leagueCountry.value_counts()
Out[13]:
Referee¶
In [14]:
referee_index = 'refNum'
referee_cols = ['refCountry']
refrees = get_subgroup(data_df,referee_index,referee_cols)
refrees.head()
Out[14]:
In [15]:
refrees.refCountry.nunique()
Out[15]:
Countries¶
In [16]:
countries_index = 'refCountry'
countries_cols = [
'Alpha_3', # rename this name of country
'meanIAT',
'nIAT',
'seIAT',
'meanExp',
'nExp',
'seExp',
]
countries = get_subgroup(data_df,countries_index,countries_cols)
countries.head()
Out[16]:
In [17]:
rename_cols = {'Alpha_3':'countryName'}
countries = countries.rename(columns=rename_cols)
countries.head()
Out[17]:
Dyads¶
In [18]:
dyands_index = ['refNum', 'playerShort']
dyands_cols = [
'games',
'victories',
'ties',
'defeats',
'goals',
'yellowCards',
'yellowReds',
'redCards',
]
dyands = get_subgroup(data_df,dyands_index,dyands_cols)
dyands.head(10)
Out[18]:
In [19]:
#get those rows where #redcards > 1
dyands[dyands['redCards']>1]
Out[19]:
In [20]:
dyands.redCards.max()
Out[20]:
Analysis¶
Players¶
In [21]:
players.shape
Out[21]:
In [22]:
players.columns
Out[22]:
In [23]:
#players name adds no value
players.drop('player', axis=1, inplace=True)
players.head()
Out[23]:
In [24]:
#Visualize the missing data on a sample of 500 records
msno.matrix(players.sample(500),
figsize=(16,7),
width_ratios=(15,1)
)
Out[24]:
In [25]:
msno.bar(
players.sample(500),
figsize=(16,7)
)
Out[25]:
In [26]:
msno.heatmap(players.sample(500),figsize=(16,7))
Out[26]:
Insight: PhotoID is correlated with the rater1 and rater2¶
In [27]:
#Check for number of missing values
print("# players: ", len(players))
print("# rater1 missing: ", len(players[(players.rater1.isnull())]))
print("# rater2 missing: ", len(players[(players.rater2.isnull())]))
print("# rater1 & rater2 missing: ", len(players[(players.rater1.isnull()) & (players.rater2.isnull())]))
In [28]:
#Get the dataframe with no nulls
players = players[players.rater1.notnull()]
players.head()
Out[28]:
In [29]:
len(players)
Out[29]:
In [30]:
msno.matrix(players.sample(500),figsize=(16,7))
Out[30]:
Now that each player has both ratings, how are combining them? But, before combinig, are they close enough to combine? How to check close enough? -> Correlations
In [31]:
fig, ax = plt.subplots(figsize=(12,10))
sns.heatmap(pd.crosstab(players.rater1, players.rater2), cmap='Blues', annot=True, fmt = 'd', ax=ax)
ax.set_title('Correlation between Rater1 & Rater2')
fig.tight_layout()
bottom, top = ax.get_ylim()
ax.set_ylim(bottom + 0.5, top - 0.5)
Out[31]:
From this, we can see that, both ther raters rated approximately the same. Therefore combine the ratings by averaging them.
In [32]:
players.head()
Out[32]:
In [33]:
players['skinTone'] = players[['rater1','rater2']].mean(axis=1)
players.head()
Out[33]:
Analyze the data distributions¶
skinTone¶
In [34]:
sns.distplot(players.skinTone,kde=False)
Out[34]:
Position¶
the position of the player might affect the redCard. like, chances of a defender getting a red might be more than a keeper
In [35]:
MIDSIZE = (12,8)
fig,ax = plt.subplots(figsize=MIDSIZE)
players.position.value_counts(dropna=False, ascending=True).plot(kind='barh',ax=ax)
ax.set_ylabel("Position")
ax.set_xlabel("Counts")
# fig.tight_layout()
Out[35]:
Create high level categories for positions
In [36]:
positions = players.position.unique()
positions
Out[36]:
In [37]:
defense = ['Center Back','Defensive Midfielder', 'Left Fullback', 'Right Fullback']
midfield = ['Right Midfielder', 'Center Midfielder', 'Left Midfielder']
forward = ['Attacking Midfielder', 'Left Winger', 'Right Winger', 'Center Forward']
keeper = ['Goalkeeper']
#Modify the DF
players.loc[players['position'].isin(defense), 'position_agg'] = 'Defence'
players.loc[players['position'].isin(midfield), 'position_agg'] = 'MidField'
players.loc[players['position'].isin(forward), 'position_agg'] = 'Forward'
players.loc[players['position'].isin(keeper), 'position_agg'] = 'Keeper'
players.head()
Out[37]:
In [38]:
MIDSIZE = (12,8)
fig,ax = plt.subplots(figsize=MIDSIZE)
players.position_agg.value_counts(dropna=False, ascending=True).plot(kind='barh',ax=ax)
ax.set_ylabel("Position")
ax.set_xlabel("Counts")
# fig.tight_layout()
Out[38]:
In [39]:
#Examine pair-wise relationships
fig, ax = plt.subplots(figsize=(10,10))
scatter_matrix(players[['height','weight','skinTone']], alpha=0.2, diagonal='hist', ax=ax)
Out[39]:
In [40]:
fig, ax = plt.subplots(figsize=MIDSIZE)
sns.regplot('weight', 'height', data=players, ax=ax)
ax.set_ylabel("Height [cm]")
ax.set_xlabel("Weight [kg]")
fig.tight_layout()
Insight: No outliers. But if we create different classes of heights and weights, can we see any pattern?¶
In [41]:
weight_classes = [
'vlow_wt',
'low_wt',
'avg_wt',
'high_wt',
'vhigh_wt'
]
players['weightClass'] = pd.qcut(players['weight'], len(weight_classes), weight_classes)
players.head()
Out[41]:
In [42]:
height_classes = [
'vlow_ht',
'low_ht',
'avg_ht',
'high_ht',
'vhigh_ht'
]
players['heightClass'] = pd.qcut(players['height'], len(height_classes), height_classes)
players.head()
Out[42]:
Pandas Profiling¶
In [43]:
pandas_profiling.ProfileReport(players)
Out[43]:
In [44]:
# modifying dataframe
players['birth_date'] = pd.to_datetime(players.birthday, format='%d.%m.%Y')
players['age_years'] = ((pd.to_datetime("2013-01-01") - players['birth_date']).dt.days)/365.25
In [45]:
players_cleaned_variables = [#'birthday',
'height',
'weight',
# 'position',
# 'photoID',
# 'rater1',
# 'rater2',
'position_agg',
'weightClass',
'heightClass',
'skinTone',
# 'birth_date',
'age_years']
In [46]:
pandas_profiling.ProfileReport(players[players_cleaned_variables])
Out[46]:
In [47]:
clean_players = players[players_cleaned_variables]
clean_players.head()
Out[47]:
In [48]:
dyands.head()
Out[48]:
In [49]:
dyands['totalRedCards'] = dyands['yellowReds'] + dyands['redCards']
dyands.rename(columns={'redCards':'strictRedCards'}, inplace=True)
dyands.head()
Out[49]:
In [50]:
#Remove records for the players who don't have skintone
dyands.reset_index().head()
Out[50]:
In [51]:
dyands.reset_index().set_index('playerShort').head()
Out[51]:
In [54]:
#Merge player & referee
player_dyand = clean_players.merge(
dyands.reset_index().set_index('playerShort'),
left_index=True,
right_index=True
)
player_dyand.head()
Out[54]:
In [55]:
clean_dyands = (dyands.reset_index()[dyands.reset_index()
.playerShort
.isin(set(clean_players.index))
]).set_index(['refNum', 'playerShort'])
clean_dyands.head()
Out[55]:
In [56]:
#Disagreedate the player-referee combo
colnames = ['games', 'totalRedCards']
j = 0
out = [0 for _ in range(sum(clean_dyands['games']))]
for index, row in clean_dyands.reset_index().iterrows():
n = row['games']
d = row['totalRedCards']
ref = row['refNum']
player = row['playerShort']
for _ in range(n):
row['totalRedCards'] = 1 if (d-_) > 0 else 0
rowlist=list([ref, player, row['totalRedCards']])
out[j] = rowlist
j += 1
tidy_dyands = pd.DataFrame(out, columns=['refNum', 'playerShort', 'redcard'],).set_index(['refNum', 'playerShort'])
tidy_dyands.head()
Out[56]:
In [58]:
tidy_dyands.redcard.sum()
Out[58]:
In [60]:
clean_dyands.games.sum()
Out[60]:
In [61]:
!conda install pivottablejs -y
In [62]:
from pivottablejs import pivot_ui
In [63]:
temp = tidy_dyands.reset_index().set_index('playerShort').merge(clean_players, left_index=True, right_index=True)
In [64]:
temp.shape
Out[64]:
In [66]:
# pivot_ui(temp[['skinTone', 'position_agg', 'redcard']], )
In [67]:
# How many games has each player played in?
games = tidy_dyands.groupby(level=1).count()
sns.distplot(games);
In [68]:
(tidy_dyands.groupby(level=0)
.count()
.sort_values('redcard', ascending=False)
.rename(columns={'redcard':'total games refereed'})).head()
Out[68]:
In [70]:
(tidy_dyands.groupby(level=0)
.sum()
.sort_values('redcard', ascending=False)
.rename(columns={'redcard':'total redcards given'})).head()
Out[70]:
In [72]:
(tidy_dyands.groupby(level=1)
.sum()
.sort_values('redcard', ascending=False)
.rename(columns={'redcard':'total redcards received'})).head()
Out[72]:
In [74]:
tidy_dyands.groupby(level=0).size().sort_values(ascending=False)
Out[74]:
In [75]:
total_ref_games = tidy_dyands.groupby(level=0).size().sort_values(ascending=False)
total_player_games = tidy_dyands.groupby(level=1).size().sort_values(ascending=False)
In [76]:
total_ref_given = tidy_dyands.groupby(level=0).sum().sort_values(ascending=False,by='redcard')
total_player_received = tidy_dyands.groupby(level=1).sum().sort_values(ascending=False, by='redcard')
In [78]:
sns.distplot(total_player_received, kde=False)
Out[78]:
In [79]:
sns.distplot(total_ref_given, kde=False);
In [80]:
tidy_dyands.groupby(level=1).sum().sort_values(ascending=False, by='redcard').head()
Out[80]:
In [83]:
tidy_dyands.sum(), tidy_dyands.count(), tidy_dyands.sum()/tidy_dyands.count()
Out[83]:
In [85]:
player_ref_game = (tidy_dyands.reset_index()
.set_index('playerShort')
.merge(clean_players,
left_index=True,
right_index=True)
)
In [86]:
player_ref_game.head()
Out[86]:
In [87]:
bootstrap = pd.concat([player_ref_game.sample(replace=True,
n=10000).groupby('skinTone').mean()
for _ in range(100)])
In [88]:
ax = sns.regplot(bootstrap.index.values,
y='redcard',
data=bootstrap,
lowess=True,
scatter_kws={'alpha':0.4,},
x_jitter=(0.125 / 4.0))
ax.set_xlabel("Skintone");
Conclusion: skintone does play the only role in red cards.¶
In [ ]: